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ABSTRACT 

In an attempt' to increase the usefulness of future 
Cflational Assessment of Educational Progress (NAEP) findings, yet 
retain the usefulness of past assessments, a framework for the 
conduct of assessments is presented . *This framework is supported by 
agreement on three major goals, by overall methods .for reaching those 
„ goals, and by firm concepts of how these goals and methods affect the 
interrelated phases of assessment work. The three most; basic purposes 
that NAEP should serve over the next five years are: (1) NAEF should 
) provide the basis for a highly credible and useful national report 
card, (2) NAEP should improve the utility of assessment results for 
educational "policymakers and practitioners, and (3) NAEP should 
improve the utility of assessment methodology to states and local 
education agencies. A discussion, on a conceptual level, indicates 
how the Educational Commission of the States believes NAEP should 
gather, process, and transmit information and how electronic 
technology can support these efforts. Generally excluded from this 
overview of assessment design are the yet to come detailed 
descriptions of specific assessment activities. Primary type of 
information provided by report: Program Description (Operating 
< Policies); Procedures (Conceptual). (PN) 
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DESIGN OF THE NATIONAL ASSESSMENT 
OF EDUCATIONAL PROGRESS 



Introduction 

Recent weeks have been a time of intense examination and re-examination 
of the design of the National Assessment of Educational Progress (NAEP). 
'Working with Research Triangle Institute (RTI) and Westinghouse Informa- 
tion Services and conferring with various outside experts, the Education 
Commission of the States (ECS) has faced the challenging and often exhil- 
arating task of setting a course that increases the usefulness of future 
national assessments yet retains the usefulness of past assessments. 

It has been in many ways a time of ferment, of wide-ranging discussions 
about new ways to meet new needs. It has also been a time of re- 
dedication, of renewed realization that — even as new needs are met in new * K 
ways — what has been uniquely valuable about the National Assessment must 
continue. 

Above all, it has been a productive time of hammering out strategies. The 
result is a solid framework for the conduct of assessments that is supported 
by agreement on three major goals, by overall methods for reaching those 
goals, and by firm concepts of how these goals and methods affect the 
interrelated phases of assessment work. 

The goals and methods are listed below. They are followed by discussion, 
on a conceptual level, of how ECS believes NAEP should gather, process, 
and transmit information and how electronic technology can support these 
efforts. Generally excluded from thi$ overview of assessment design are 
detailed descriptions of specific assessment activities; these activities will, 
of course, be discussed thoroughly in ECS's grant application submitted to 
the National Institute of Education later this month. 

Goals 

The three most basic purposes that NAEP should serve over the next five 
years are these: 

< 

• NAEP should provide the basis for a highly credible and useful 
national report card. 

• NAEP should improve the utility of assessment results for educa- 
tional policymakers and practitioners. 

• NAEP should improve th^r utility of assessment methodology to 
states and local education agencies. 



Methods for Reaching These Goals^ X 

The goals are simply stated. But reaching them will not be simple. To 
guide their efforts in the coming years, ECS, RTI, and Westinghouse have 
therefore further agreed that the following nine considerations should be 
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paramount in the design of the assessment. All measures adopted, plans 
made, and actions taken should: 

• Avoid leading to the development of federal tests, standards, or 
curricula. 

• Use a national consensus approach that reflects the diversity of 
American society. 

• Maintain strong, cooperative relations with schools. 

. a 

• Take into account future needs for information on educational 
progress. 

• Increase the quality of assessment methodology. 

J • Provide comparability with previous assessments so that trends in 
education can be measured. 

• Complement— but not replace— state and local assessment efforts. 

• Make good us6 of technology. 

• Use other resources efficiently. 



Purposes Served by Goals and Methods 

These goals and methods have served a double purpose in the process of 
designing a National Assessment that can rise to the complex challenges of 
the next five years. First, they emerged from the sustained effort to set 
priorities. That is, they resulted from the process of deciding which issues 
are of overriding importance. Then, they were built back into the design of 
the assessment work, becoming the basis for making further decisions. 

The information-gathering phase is described below in two sections, one on* 
Assessment Development and one on Data Collection. Analysis outlines 
plans for the efficient processing of information. Assessment Utilization 
treats the three major aspects of transmitting information: reporting, dis- 
semination, and technical assistance. A section on Use of Electronic Tech- 
nology describes examples of improved ways to manage NAEP's information 
resources. 

Successful assessments cannot be designed as a series of separate phases. 
Since all three major goals lead to improving the usefulness of NAEP, ways 
to bring about that improvement are built into each phase. 

Assessment Development 

In designing assessments, NAEP must balance three key considerations: the 
relevance of what NAEP assesses, the validity of its methods and results, 
and the usefulness of both methods and results for a variety of purposes. 

It must, further, base the development of assessments on these primary 
assumptions: 

1. Because the National Assessment is^a public enterprise, it should 
inyolve a wide range of people in its development activities, using 
a consensus approach to developing its objectives. ^ 
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2. Although the assessment items must meet the highest psycho- 
metric standards, assessments are not norm-referenced tests and 
procedures useful in other kinds of testing should be introduced 
only after considerable thought and research. 

3* The Assessment must continue to develop exercises to assess 
important higher-order skills, resisting the temptation to depend 
totally on multiple-choice items, which is a false economy. 

4. Assessment subject areas can be integrated in ways that enhance 
the usefulness of findings and do not endanger continuity or 
validity. 

5. It has become increasingly important to gather background infor- 
mation about achievement-related variables that can be altered. 

6. A collaborative relationship between NAEP and state or local 
assessments has mutual technical and financial advantages. 



The student performance assessment development framework. ECS 

believes that NAEP should develop assessments that integrate several subj- 
ect areas, thereby (1) responding to widespread interest in topics that span 
subject areas, (2) consolidating into the development of single sets of items 
resources that were earlier spread across several learning areas, and 
(3) generating exciting new kinds of information about education. A new 
\ framework incorporating communication skills, basic knowledge, and think- 
ing skills can be used to guide decisions about which objectives and 
achievement items should be included in each integrated assessment. 
Table 1 shows the framework for developing achievement objectives and 
items in integrated assessments. 



\ The background variables framework. ECS recommends that NAEP 
collect more information on achievement -related variables and somewhat 
different information, emphasizing particularly the acquisition of data 
about factors and conditions that are amenable to change. Currently, 
extensive research is being conducted on school effectiveness. Assess- 
ments should therefore collect information about achievement -related var- 
iables that will help people interpret this research and increase the policy 
relevance of research findings. It will be especially important to collect 
information of^TTTe^S^ct that will aid people now working in school- 
improvement arograms.^ Table 2 shows a framework for developing 
achievemGnt-reffcted variables. 

The consAgus process. Consensus, defined as a carefully-negotiated 
general agreement about goals, has been and will continue to be vital to the 
development of national assessments. But ECS proposes to streamline pro- 
cedures for reaching consensus by drawing, where appropriate, on the vast 
experience it has already gained, on materials that are already available 
from outside sources, and on time-saving, cost-effective new techniques of 
communication like computer networking. 
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/ Improved measurement techniques. ECS anticipates continuing the 
j gradual modification of measurement techniques in response to changing 
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Table 1 

< PROPOSED LEARNING AREA DEVELOPMENT MODEL FOR NAEP 



SKILLS 



Communication 

• Receiving language and 
symbols (ways to gain 
information) 

Observing 

Listening 

Reading 

• Producing language 
and symbols 
(ways to transmit 
information) 

Showing (gesturing) 
Speaking 

Graphing/drawing 

Calculating 

Writing 



Thinking 

Recognition/recall 

Comprehension 
(understanding) 

Application 
(abstraction) 

Analysis/synt hesis 

Evaluation 



KNOWLEDGE 
of Disciplinary 
Content 

• Specific facts 

• Conventions 

• Main ideas 

• Procedures 

• Concepts 




Table 2 

PROPOSED ACHIEVEMENT-RELATED VARIABLE 
DEVELOPMENT FRAMEWORK FOR NAEP 



STUDENT LEVEL 
AH Booklets 



•Age 

•Race/ ethnicity/ 
•Sex / 
•Parents 1 education 

Parents 1 occupation 
•Language in hpme 
•Birth order i 
•Reading materials in 
the home / 

(17s onW) 
•High school program 
Employment] 
aspiratk 



r 



/ 



/ 



Matrix Sample 



SCHOOL LEVEL 
School Program Questionnaire 
All Schools 

•Size of community 
•Type of community 
•Public/private 
•White 

Minimum competency testing 

Total enrollment 

Days in school year 

Minutes per class period 

Required classes per day 

Amount of time for non- 
academic instruction 

Average absenteeism 

Bused for racial balance 

Type of high school 
(vocational/academic) 

Average class size 

Amount of parent involvement 

Discipline 



Department Head or Curriculum 
Coordinator Questionnaire 
Matrix or Special Suboample 



/•TV viewing 
•Homework 
School spirit 
Positive feedback 
Teacher quality 
Teacher organized 
Tutoring 

Enrichment classes 

Remedial classes 

Self concept 

Attitude towards school 

Recitation opportunities 

Amount writing for classes 

Health 

Mobility (years same house, county) 

Home computer 
•Compensatory education 

Access to school computer 
•Content-specific items 

Courses taken in subject matter 

Electives taken in subject matter 

Grades received 

Teacher's methods in subject matter 
Materials used in subject matter 
courses 

* 



Principal as education leader 
Objectives developed jointly 

(teachers and principal) 
School pride, spirit 
High expectations 
Discipline 
Time-on-task 
Emphasis on basics 
In-service educational opportunities 
Amount homework assigned 
Non-qualified special teachers 

(math, science, reading, writing) 



•Currently collected. 
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expectations, needs, and technological possibilities. Balancing the effici- 
ency of using easy-to-score multiple choice items witTi the need to measure 
higher-order and problem-solving skills remains important, as does preserv- 
ing NAEP's ability to monitor trends. NAEP should be designed to ensure 
that NAEP items measure what they purport to measure, allow reporting of 
skills through organizing items in subsets measuring common domains, and 
make possible the cautious introduction of procedures for scaling items 
(like item response theory). Over the longer term, explorations, of such 
possibilities as computer-assisted testing should continue. 

TThs 1983-84 assessment of reading and writing and assessments 
thereafter. ECS believes NAEP should undertake to integrate, during the 
1983-84 school year, the writing assessment required by legislation in 1983- 
84 and the reading assessment required in 1984-SS^ For reasons summar- 
ized above in the discussion of the student performance development 
framework, such an integrated assessment is timely and feasible. ECS 
further believes NAEP should field an integrated mathematics/science/' 
technology assessment in 1985-86. 

Although in general this report does not describe specific activities that 
will be undertaken to carry out integrated assessments, one proposed pro- 
cedure should perhaps be mentioned here because it represents a significant 
change from past practice. To reduce printing and scanning costs, and to 
make it easier for states to gather N AEP-comparable data, ECS believes 
NAEP should replace machine-scannable booklets, which are very expensive 
to design and print, with a combination of scannable answer folders and 
separate (reusable) assessment booklets. Over the longer term, exploration 
of the possibilities of direct data entry for scoring should continue. 

Data Collection 

Data collection should continue a tradition of sound methodology, accurate 
measurement, valid statistical samples, and excellent cooperation by 
schools. Some alterations of proceditfe are necessary in the interests of 
lowering costs and of aiding other aspects of assessment design, like the 
change to integrated assessments covering several subject areas simultan- 
eously. Three specific changes are particularly important. 

• To maintain the comparability of old and new data, the National 
Assessment should continue to measure the performance of about 
60,000 9-, 13-, and 17-year-olds enrolled in public and private 
schools. But sophisticated analytical techniques cata be developed 
that allow NAEP data to be compared to grade-level data collected 
by state and local education agencies. Although the generation of 
grade-level data will be primarily a matter of analysis (sec Anal- 
ysis) , it will also" involve collecting data at the eighth-grade level 
with a sample of 3,600 students in the 1983-84 assessment. 

• Data should be collected, in alternate years, in three equivalent 
ses5Tfcns of six to eight weeks held between November 1 and 
Aprilfi^. All three age groups will be assessed during each of these 
sessions. The advantages of collecting data in fall, winter, and 
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spring sessions are several: costs are lower, the burden on school 
distrj^Jts is reduced, analysis potential is enhanced, the ability to 
measure and report trends is maintained, and tfte correlation of 
national with state results is simplified. 

• A variety of assessment booklets should be administered to each* 
group of students during a single session, with a field administrator 
replacing pacetl tapes. This will reduce costs, maximize item and 
content coverage, and maintain precise national performance 
estimates. 

Another important aspect of data collection design has been mentioned in 
the preceding section: the use of separate machine-scannable answer 
folders. Studies should be built into the 1983-84 assessment to allow 
calibration with past data collection practices. 



Analysis 

\? 

Drawing conclusions about achievement from masses of raw data presents 
challenging problems that can bp resolved in two btfsic ways. NAEP staff 
should continue to undertake analysis (of the sort outlined below), and data 
should be made not only available but also increasingly accessible to out- 
side researchers interested in performingtheir own analyses. 

ECS believes NAEP should continue to use percentages of acceptable 
responses as its primary indicator of achievement. That is, national 
assessments should continue to be objective-referenced rather than norm- 
referenced. It should continue to present summary-level estimates of 
achievement for age groups as, a whole and also for various subgroups— for 
repeated exercises that allow the measurement of change from one assess- 
ment to the next and also for new exercises designed to measure new 
skills. It should continue to use a national probability sample at each age 
level. It should continue to deal cautiously with the complex interrelation- 
ships of data, basing its methodology for interpreting relationships on 
weighted class adjustments, Tukey's balancing analysis, correlation anal- 
yses, and general linear model techniques. 

In three other areas of analysis, ECS proposes that NAEP break new 
ground. 

• NAEP should undertake major research to determine the usefulness 
of Item Response Theory (IRT) in assessing achievement and change 
in achievement over time. IRT models, which use mathematical 
functions to relate the probability of success on an item to the 
ability measured by the test that contains it, are still highly con- 
troversial. But they are also promising in many ways. NAEP 
should proceed carefully with a parallel analysis for the 1983-84 
assessment that will allow comparison of IRT-based results with 
traditional percent-cojrect results. 

• NAEP should actively pursue ways to rielate NAEP data to other 
data bases and studies. Some potential relationships are direct, 
like the correlation of NAEP data with census summary tapes. 
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Others are indirect, like those that can be established using meta- 
analysis techniques and data from large-scale educational studies 
like the High School and Beyond study sponsored by the National 
Center for Educational Statistics. 

• NAEP should explore ways of providing estimates of achievement 
fdr grades, 4, 8, and 11 as well as for ages 9, 13, and 17. Since 
NAEP already gathers data for age groupings as well as for single 
ages (assessing the abilities of 9- and 13-year-olds on some of the 
same items, of 13- and 17-year-olds on some, and of 9-, 13-, and 
17-year-olds on others), it will be possible — through use of a care- 
fully designed conversion formula— to interpolate results for grades 
as well as ages. To check the validity of the formula, a sample of 
eighth-grade students (including 900 who are not 13 years old) 
should be assessed in 1983-84. If grade results can be derived from 
age data, data in future years can be reported both ways. 

If a degree of caution has been apparent in the foregoing discussion of 
recommended analytic strategies, ECS considers this only appropriate. For 
conducting national assessments is a sensitive matter, and the value of 
assessment results is all too easily jeopardized by incautious experimenta- 
tion. Rushing to judgment seems perhaps least advisable in the matter of 
interpreting the significance of complex and, in some senses, fragile data. 
Yet, understandably enough, education policym ackers, and members of the 
general public want to know not simply what the data are, but what they 
mean. 'ECS therefore^recommends a two-part strategy. NAEP itself should 
continue to exercise reasonable caution in (frawing conclusions based on 
analysis of data. But outside experts— acknowledged leaders in education 
representing diverse points of view— should be invited to draw thqir own 
conclusions about significance. This strategy, which bridges the analysis 
and reporting phases, is discussed further in the following section. 



Assessment Utilization 



Basic to the design of the National Assessment is the conviction that the 
utility of assessments cannot be effectively developed after the fact but 
mast instead be an integral consideration in all phases of assessment 
work. Broadening the consensus process to include potential users of NAEP 
information, improving coverage of learning areas and skills, increasing 
emphasis on alterable variables,* providing grade-level results as well as 
age-level results, combining NAEP data bases with other bases— these are 
only some of the ste£>s that can help increase the usefulness of national 
assessment results and methods to the many different groups concerned 
about educational progress. 

ECS recognizes, however, that developing, collecting, and analyzing useful 
information is not enough: information must also be packaged appropri- 
ately and disseminated to the right people at the right time. 



•Variables amenable to change through policy decisions and instructional 
approaches. 




NAEP conveys two*major types of information — results, defined as the find- 
ings, analyses, anjd interpretations resulting from NAEP's^data collection 
^ efforts, and methods, defined as the features of the assessment methodol- 
V egy itself that others adopt as they undertake aissessments. 

Analysis of NAEP use indicates that results are used most heavily at a 
national level, methods most frequently at the state level. NAEP results 
areifrequently presented and interpreted in national education newsletters, 
journals, and conferences they often serve as background and context, 
informing public debate and the 'exploration of policy options; they are used 
v to plan textbooks. NAEP methodology has served as the model fo> many 
state assessment programs. 

Clearly, new strategies are needed to ensure the use of NAEP results at the 
state * and local levet The collection of achievement-related variable 
information during the assessment^ one important strategy, but changes in 
reporting,' dissemination, and technical assistance are also essential. 

Reporting. For the next f*w years, NAEP reporting strategies should 
emphasize disseminating NAEP results to state and local education decision 
makers, since NAEP methods are 9 already relatively familiar to state and 
district testing directors. t 

4 

To help educators, policymakers, and the public understand not only where 
things stand in American education but also how things got that way, ECS 
jreports should emphasize the relationships between student performance 
and alterable practices. A key to doing this successfully is to report NAEP 
findings within the context of evidence from other sources, adding conflict- 
ing and supportive data from other studies to interpretations of NAEP data 
provided by leading educators. 

Significant improvement is needed in "packaging" strategies that tailor the 
medium in which information is presented to the needs of particular audi- 
ences. For example, reports should b£ prepared for curriculum specialists 
arid textbook selection confWiittees that describe results for learning areas 
and specific skills; analyses of policy issues for legislators, chief state 
school 6f€icers, governors and their aides should relate NAEP results 
directly to policy in easily-used summary form. 

ECS recommends using advances in technology to begin preparing an infor- 
mation data base containing NAEP results in different forms, identified by 
intended audiences. For example, a NAEP report currently4>eing prepared 
by ECS summarizes the implications for teaching and testing of findings 
based on three repeated assessments in writing, reading; and mathe- 
matics. This report contains the essenti# information for chief state 
school officers, legislators, textbook selection committees, ahd local edu- 
cational administrators, but many of the people may not wisn— or need— to 
read the entire report. If the report was entered on word processors at 
regional information clearinghouses and state educational agencies, how- 
ever, excerpts could be made available to a school board, say, or to a local 
administrator. ECS could also develop tailored reports. So could other key 
information resource centers, with technical assistance from NAEP. 

i 
i 
i 



Another important reporting approach is to increase access to user tapes so 
that larger numbers of researchers can conduct more independent analyses 
that illuminate educationakpolicy. 

f 

Dissemination. The cornerstone of the dissemination effort should be 
a systematic investigation of decision-making processes to identify key pol- 
icy groups and individuals. Thereafter, NAEP should emphasize three dis- 
semination strategies: 

1. Build on existing networks and information-brokering systems, 

2. Make members of NAEP audiences active participants rather than 
passive consumers by including them in assessment design. 

3. Promote NAEP products vigorously. 

If ECS administers NAEP, networking holds great promise of success for a 
number of reasons. ECS is already widely recognized as a reputable broker 
of information with broad connections not only to the education community 
but also to policymakers, particularly at the state level. Knowledge utili- 
zation theory has confirmed the vital importance of networks as sources of 
information— especially for decision makers who frequently turn to 
informal contacts for new ideas and information. Linking with other net- 
works aligns ECS to leverage its own efforts, reaching points from which 
information is distributed to teachers, administrators, board members, and 
parents, for example, rather than having to reach each individual member 
of these audiences. Networks are efficient, in the sense that the people 
who take part in them generally do so because they are already interested 
in finding and using information. 

Encouraging members of NAEP audiences to become active participants in 
various phases of assessment work has two primary purposes, improving the 
usefulness of assessments and also their actual use. When, for example, 
members of a curriculum organization help develop assessments, they have 
a part to play in deciding which achievement-related variables are most 
relevant to curriculum concerns. Then, no longer outsiders presented with 
unfamiliar results but instead insiders who have helped shape those results, 
they can be encouraged to do secondary research, pass results along to 
LEAs and teachers, and, in general, increase the use of national 
assessments. 

To strengthen its promotional efforts, NAEP should pay particular atten- 
tion to it3 ties with media that, in turn, disperse information to very broad 
audiences. Special efforts should be made to establish better links with the 
electronic media like the Public Broadcasting System and National Public 
Radio. Direct dissemination efforts; like publication of the quarterly 
NAEP Newsletter (which now reaches more than 36,000 recipients) and of a 
quarterly bulletin for NAEP data users are also important and should 
continue. 



Technical assistance. Technical assistance has generally played a 
more important role in supporting the utilization of NAEP methods than of 
NAEP results. ECS recommends shifts in NAEP f s technical assistance 
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activities. Providing services to individuals is sometimes essential- 
responding to a telephone inquiry, writing an explanatory letter, doing a 
special data run. But, whenever possible, NAEP should (l)use materials 
that have already been prepared (printed or pre-packaged materials, peri- 
odic bulletins) and (2) provide services to groups rather than to individuals, 
concentrating especially on training grpups of natural service providers 
that can in turn train others in the; use of NAEP methods. Training the 
trainers has worked well in the past, and it is an approach that allows 
NAEP to leverage limited resources. 

NAEP should also shift emphasis toward giving technical assistance in dis- 
semination to key information providers at the $t#te level. Another critic- 
ally important service: providing state assessment directors with models 
they» can use to disseminate assessment results. 



Use of Electronic Technology 

Tools to improve the use of information resources— hardware, data bases, 
information systems— are essential to the conduct of all phases of national 
assessments. NAEP needs to contine increasing its effective use of infor- 
mation resources. It should also continue exploring promising opportunities 
to incorporate technological advances that will aid the conduct of future 
assessments. V 

Selected ways of improving the management of NAEP information 
resources to support a strong National Assessment are outlined below, as 
are three areas in which the use of technology can be enhanced. 



Assessment development support. Continued automation of the devel- 
opment of items and objectives would be highly useful, so that .NAEP staff 
can review and revise variables interactively. Automated packaging of 
items and exercises will increase efficiency. 



Support for analysis. Continued development of a data base manage- 
ment system that offers rapid response, flexibility in data organization, 
security safeguards, and access from a variety of perspectives for multiple 
users is critical. Simple-to-use, on-line methods of access that allow users 
to sort and query a variety of data bases will expand greatly the analyses 
that can be done, thus ^hancing the reporting of useful results. 

Utilization support. To improve the dissemination of information, 
direct on-line acaeSs to data should be provided. Indirect access through 
public use data tapes should continue. 



Enhanced technology. Research in three areas holds special promise 
for incorporating new technology that will expand qr improve the capabil- 
ities of NAEP. 

Developing microcomputer application systems could greatly improve 
access to NAEP data. The relatively low cost of microcomputers and their 
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expanding capabilities mean that they are fast becoming widely used 
tools. It therefore seems^ appropriate to consider microcomputers as a 
potentially very useful system for giving local and state education agencies 
access to NAEP items and objectives. - / 

Modeling and simulation programs could help state and local education 
agencies and secondary researchers understand the complex relationships 
between educational progress and educational curriculum, environmental 
factors, social factors, and budgeting decisions. Another potential use of 
models, which could be explored, would be to link national assessment data 
with state and local data. 

The time may be fast approaching for computer-assisted testing, which 
allows students whose educational progress is being assessed enter their 
responses directly into a computer. Direct data entry may offer an effici- 
ent and cost-effective way to enter and score higjv-quality data. Thorodfch 
exploration of the advantages and disadvantages of these options seems 
appropriate at this point. 



Conclusion 

Table 3 contrasts, in summary form, current NAEP practices with rjew 
measures designed to preserve trend analysis capability while improvihg 
cost-effectiveness and enhancing utility. 

When ECS applied for a contract from the National Institute of Education 
to support development of a design for the National Assessment of Educa- 
tional Progress, the Commission described in some detail the process it 
should use to create a design* Although some aspects of design continue to 
develop, that process is now essentially complete. Task forces have met, 
conferred, drafted reports, revised reports, conferred again. Outside con- 
sultants have contributed broader perspectives in some instances, special- 
ized knowledge in others. ECS management have choreographed meetings, 
conferred with colleagues at RTJ and Westinghouse, reviewed reports, and 
evaluated options. 

One aspect of the process deserves special mention here, because ft has had 
a very direct effect on the design of the national assessment described in 
this report: even as ECS was designing a National Assessment that would, 
meet the needs of the future, it continued to work on the National Assess- 
ment in 1982. 

For ECS, the opportunity to stand back from current activities— and to look 
ahead to broad future prospects— was very welcome. For it made possible a 
distancing from the day-to-day activities of the current assessments and 
helped to clarify priorities and bring agreement on major goals. For NIE, 
the advantages are somewhat different. For the fact that ECS f s assess- 
ment design was developed in the context of on-going assessment work 
means that the framework ECS proposes is supported by years of direct 
experience and solidly grounded irreality. 
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TaMc 3 

BUILDING A NATIONAL ASSESSMENT FOR THE 19t0si SUMMARY OF CONTRASTS BETWEEN PAST AND FUTURE PRACTICES 



iUMment Areas 



Past Practices 



Future Practices 



Enhancement Designed to: 
Improve kicrease Preserve 
Utility Efficiency Trend Data 



1. Assessment Cycle Timeline 



1. Assessment Development 

a. Learning areas assessed 

b. Objectives and item 
development process 

c. Assessment items 

d. Assessment-booklets 

e. Background variable 
data collected 

f. Background variable 
collection procedures 



g. Total number of assess- 
ment booklets 

h* Criteria for objectives 
inclusion 



Total cycle required 57 
months, with first report 
available 12-15 months 
after end of data 
collection 



One or two learning areas, 
assessed as distinct, sep- 
arate areas 

National consultants and 
lay reviewers 

Multiple-choice and open- 
ended items 

Students respond in actual 
assessment booklets 

Demographic, non-alterable 
variables emphasized 

Generally included in every 
assessment booklet admin- 
istered to an age group for 
the learning area 

Approximately 40 

What is taught in schools 
as well as what is needed 
for future 



Total cycle requires 48 
months, with first report 
available 4-6 months after 
end of data collection 



Multiple learning areas, 
assessed as an integrated 
area of learning 

Similar, but streamlined by 
greater building on past con- 
sensus 

Refined items to measure 
higher-order cognitive skills 

Students respond on separate 
answer folders 

Achievement-related variables 
dealing with alterable situa- 
tions or practices emphasized 

Some achievement-related vari- 
ables will be sampled (not in- 
cluded in every booklet); others 
will be collected by trained 
field staff 



Approximately 90 



Similar, with emphasis on what 
should be learned 
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BUILDING A NATIONAL ASSESSMENT POR THE 1910s: SUMMARY OF CONTRASTS BETWEEN PAST AND PITTURE PRACTICES 

(Continued) 



Assessment Areas 



Past Practices 



Future Practices 



Enhancement Designed to: 
Improve Increase Preserve 
Utility Efficiency Trend Data 



3. Data Collection 

a. Populations sample 



b. Data collection 
schedule 



c. Sample design 



d. Number of schools 
sampled per assessment 
booklet 

e. Number of students 
sampled per assessment 
booklet 

f. Number of students sur- 
veyed per field admin- 
istration 



g. Field staff 



h. Standardization 
procedures 



9-, 13-, and 17-year-olds 



13-year-olds In the fall; 
9-year-olds in the winter; 
and 17-year-olds in the " 
spring 

Three-stage, stratified 
probability sample with 
matrix sampling of iten)s 
and students 

Between 75 and 125 schools 
depending upon sample size 
for each assessment booklet 

Between 1200 and 2500 
students 

Between 16 and 25 students 
all responding to the same 
assessment booklet 

Specifically trained and 
hired; school staff not 
required to administer 
assessment booklets 

Paced audio tapes used for 
directions and pacing 
students 



9-, 13-, and 17-year-olds plus 
8th grade students who are not 
1 3-year-olds 

All 3 ages assessed at each 
of the 3 times (fall, winter, 
and spring) with equivalent 
samples for each time period 

Same, with sample divided into 
equivalent thirds 



Approximately 300 schools 



Approximately 550 students 



Up to 50 students per session, 
responding to different 
assessment booklets 

Same, with field staff 
further used to collect 
achievement-related vari- 
able information 

Field staff will present In- 
structions and pace students 




BUILDING A NATIONAL ASSESSMENT FOR THE 1910s: 



Table 3 

SUMMARY OP CONTRASTS BETWEEN PAST AND PUTURB PRACTICES 

(Concluded) 



Assessment Areas 



4. AmlysU 

a. Item level data 



b. Summary level'dat a 



c„ Reporting variables 



d. 



Linkage to other data 
bases 



e. Secondary research 



f. Types of analyses 



Past Practices 



Putisrc Practices 



Enhancement Designed to: 
Improve Increase Preserve 
Utility Efficiency Trend Data 



Presented for nation, sex, 
region, race/ethnicity, 
level of parental educa- 
tion, size and type of 
community, and grade 

Mean percentages and 
standard errors on objec- 
tives or item clusters 
for nation, region, sex, 
race/ethnicity, level of 
parental education, size 
and type of community, 
grade, and other background 
variables 

Focus on demographic and 
sociological variables 
above 



Informal and infrequent 

Public use data tapes 
provided to encourage 
secondary research 



Percentages 



Presented for nation, sex, 
modal grade, and perhaps 
some of the achievement- 
related variables 

Same, with greater precision 
plus achievement related 
variables 



Pocus on achievement-related 
variables, though demographic 
and sociological variables 
will be initially used for 
reporting change 

Planned and frequent 

Same, with increased emphasis 
and support plus exploration 
of on-line access for secondary 
users 

Same, plus increased emphasis 
on multivariate analyses 
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